Euroscene 2

home *** CD-ROM | disk | FTP | other *** search

/ Euroscene 2 / Euroscene 2.iso / USEFUL / libs / FileSys / XFH / XFH.doc < prev

Wrap

Text File | 1995-11-01 | 19.7 KB | 390 lines

Xpk File System Handler v1.12 By Kristian Nielsen This package may be distributed freely for non-commercial purposes along with Xpk or on its own, provided that all files and subdirectories are left intact. However, all rights remain with the author. And as usual, the author will in no way be held responsible for any damage, directly, implicit or otherwise, resulting from the use of this program. After all, you get this program for free; use it or not as you like, and at your own risk. The file 'README' (which should be distributed alone with this file) gives information on how to get started quickly while this text gives a more technical discussion of the XFH file system handler. --- O --- What's possible. XFH, when used with Xpk, makes it possible to store data in compressed format without this being visible to the user or to application programs - XFH will make the compressed data appear like ordinary files. This is by no means a new idea. In MS-DOS world, a (commercial) program called 'Stacker' has been available for some time, which makes it possible to compress a whole partition on a harddisk. On the amiga, people have long used powerpacker along with programs like PPMore to store data files in packed formats (but relying on the application to recognise that the file is in compressed format), and programs like PPPatch have been used to change dos.library to recognise the powerpacker format automatically. However, none of these approaches are perfect in all situations, and so there is room for another alternative - the XFH. XFH works in conjuction with the Xpk approach to data compression. This in itself gives a number of advantages - a flexible interface to the compressing algorithms, lots of different packers available with the possibility to add new ones etc. But compared to approaches like 'Stacker', there is an additional advantage: access to the compressed data is not limited to XFH - most of the time, the XFH way will be the most convenient way to access data, but if needed, the complete range of Xpk applications is available to the user. The file orientated nature of Xpk also means that XFH - like PPPatch etc., but unlike Stacker - will co-exists nicely with any other Amiga filing system without the need to set up any new partitions or prepare the disk with a special program. In fact, if you happened to have a CD-Rom (read only) stuffed with powerpacked files (or any other format that the current version of Xpk supports), you could dump an XFH unit on top of it and largely forget about the disk being compressed from then on. So, what is possible is to mount a XFH unit on top of each of one or more standard AmigaDOS directories. It is now possible for programs to access the directories as usual, data being decompressed and optionally compressed as needed in a completely transparent way - programs will never know the difference between compressed and uncompressed files - while still having access to the compressed data using conventional compress/uncompress programs. So the dream is that of doubling the capacity of your hard- or floppydisks for zero cost. As an example of the possibilities of the XFH, I have been using it for a couple of months for holding various doc files, metafont sources and little used emacs scripts and shell commands on HD, saving in the order of perhaps 60% file space with no noticeable degrading of overall system performance. --- O --- What is the catch? Of course, as everyone knows, nothing comes for free. Obviously, there is a speed penalty to compressing or uncompressing a file. This speed penalty will be highly dependent on the actual compression algorithm and data media used; for example an algebraic compression scheme used with a superfast HD will probably be rather slow, while a fast algorithm may actually result in a speed-up of floppy access since the disk access time saved by the smaller file size more that accounts for the time taken decompressing. And of course with the coming of ever more powerful CPU's the speed will be less of a problem. Some things will always be slow, though. Especially directory listning is a problem, since every single file has to be opened in order to check whether the file is compressed or not. In fact, I'll admit that running XFH from floppy on an unaccelerated amiga will sometimes seem a bit slow. However, XFH was not written to be as fast and small as possible, but rather to be flexible and expandable. And I'm sure a lot of users will find it very useful even on 'small' amigas. On an A3000, XFH runs like a dream, of course. And there is still the possibility for speed improvements in later versions. Aside from the problem of speed, V1.12 of the XFH comes with a few other limitations that I'm hoping to remove in later versions. Most important is the lack of most of the new 2.0 packets, a lack that will become more severe as more 2.0-only programs start to depend on these packets. Another problem is that XFH in the present version is somewhat memory-hungry, in that it will keep any compressed file completely unpacked in memory as long as that file is open. This is mostly due to limitations in the current Xpk interface, and I'm hoping to remove this in a later release. However, currently this means that it is impossible to open a compressed file if it is larger than available memory. A number of other problems to look out for are detailed in a later section in this doc file. Most of these are things that could be fixed in later versions. --- O --- Instructions for use. XFH is implemented as an AmigaDOS device handler. The 'L:' directory contains examples of other such device handlers, and this is usually also the best place to put XFH (the file 'XFH-Handler', to be precise). Like other handlers, XFH must be mounted before it can be used. This can be done by creating an entry for it in the 'DEVS:Mountlist' file (just the mountlist for short) and issuing the command 'Mount <device>', where <device> is the name given to the device in the mountlist. The supplied file 'Devs/Mountlist' contains four example entries for the XFH; it might be convenient to append this file to the end of 'DEVS:Mountlist'. The most important part of each mountlist entry is the unit number (the number used in the 'Unit = ...' line), since this is used to control what files that particulary XFH unit should (de)compress, how it should do it etc. Further information on the mechanism of mountlists and handlers in general can be found in various places; I will not attempt any lenghty explanation here. Unlike most other 'normal' device handlers, XFH needs additional information to function correctly. Most importantly it needs to be told what directory it should use as its root directory (that is, where it should look for compressed files). In the simplest case, this can be done by assigning 'XFH<n>:' to this directory, where <n> is the unit number as given in the mountlist. However, XFH also provides 'options' which provide much more flexibility and which MUST be used if the handler is to compress files automatically as they are written by applications. In the current version all options must be set in files (hopefully a later version will also provide an AREXX interface). An option is specified on a line of its own, and follows the same conventions as the TOOLTYPES of the Workbench. XFH has the concept of primary and secondary option files. The primary option file, for XFH unit number <n>, is placed in the file 'S:Xfh/.xfhrc_<n>'. The secondary option file is named '.xfhrc' and is placed in the root directory of the XFH unit (uncompressed!). The settings in the secondary option file overrides any settings in the primary option file, but there are some restrictions on the options that can be used in the secondary option file, see below. Both files are optional, the handler will pick default values for any options that are not set by the user. The supplied directory 's/xfh/' contains example primary option files for units 1-3. Boolean options can be specified with "NO" / "OFF" or. "YES" / "ON". String options are specified by simply putting the string after the 'OPTION=' bit, no quotes are needed. The available options are detailed below: ROOTDIR VOLUMENAME AUTOCOMPRESS PACKMODE STEPDOWN PASSWORD XPKPRIORITY TRUNCATEONPACK Option ROOTDIR: This option is used to set the name of the directory that XFH: is to reside in. For example 'ROOTDIR=Work:xfh' would cause XFH: to use that directory as root. Note that this option can only be used in the primary option file 's:xfh/.xfhrc_<unit>'. When used elsewhere it will simply be ignored. If this option is not specified, XFH will use the assign 'XFH<n>:', where <n> is the unit number. This is mainly for compatibility with earlier versions of XFH: ; using the ROOTDIR option is a much cleaner way of doing this. Option VOLUMENAME: This option is used to set the name of the XFH: volume. This is the name that will be used in absolute path specifications, as well as the one returned by Info(). Ie. if you go 'VOLUMENAME=Manuals', access can be by 'Manuals:' as well as by 'XHn:' (or whatever). Like ROOTDIR, this option can only be used in the primary option file. If this option is not specified, the default is to use the name of the directory in the underlying file system, or (if this is itself the root of a volume) to use the name with 'XFH_' prepended. Again, this is mainly for compatibility with early versions of XFH:. Note that this option can be seen as a replacement for the 'Relabel' command (which isn't supported). Option AUTOCOMPRESS: This options tells whether XFH: should attempt to compress the files written through it. When this option is set, everytime a file written to the XFH: is closed, an attempt will be made to compress the file to a temporary file using Xpk. If this is succesfull, the temporary file will be renamed to the original name and the uncompressed file will be deleted. If the compression fails for any reason, the uncompressed file will simply remain intact. This also means that if anything should go wrong during compression (like a disk full error), it is unlikely that any data will be lost since at least one of the two files should be intact (though possibly with a wierd name). This option is OFF by default. Option PACKMODE: This option selects the mode that Xpk should use when compressing files. It is specified in the usual way when using Xpk. For example, to use 12-bit BLZW compression, 'PACKMODE=BLZW.12' would be used. The default is to use the NUKE compression ('PACKMODE=NUKE'). Of course, to use a specific compression method, the nessesary sublibrary must be available in LIBS:. Option STEPDOWN: This option controls the Xpk flag 'XPK_StepDown' during packing. If set, it means that Xpk is allowed to reduce packing efficiency if nessesary to save memory. Refer to the Xpk documentation for details. Default is OFF. Option PASSWORD: This option is used to set the password that XFH: should pass on to Xpk when compressing or uncompressing. It should be noted that there is no attempt to keep this password safe from 'memory peekers'. Of course, storing the password in the option file isn't actually safe either. The latter problem will be solved when XFH gets an AREXX interface; until then, a script something like this copy xfhoptions/.xfhrc xfhdir: ; xfhdir: is the root of xfh: echo >> xfhdir:.xfhrc "PASSWORD=..." mount XFH: info >nil: xfh: ; Make sure it's really loaded. copy xfhoptions/.xfhrc xfhdir: echo >> xfhdir:.xfhrc "PASSWORD=***" ; Really erase from disk. could be used, though it still isn't completely safe. Finally note that if AUTOCOMPRESS is requested, files may still be saved unencrypted in low-memory or low-diskspace situations, and (depending on the underlying file system / disk device) part of the unencrypted data may still be physically stored on the disk after the deletion of the unencrypted file. If an attempt to open a file is made when XFH has been given the wrong password, the open will fail with error code 212 (Object wrong type). Option XPKPRIORITY: This option is used to set the task priority that should be used when doing Xpk operations (compress/uncompress etc). Setting this to zero or less will prevent XFH from stealing all CPU-time from tasks running at a 'normal' priority. Note that it is possible to set the priority of the handler itself in the mountlist. If this option is not used, the handler will keep whatever priority it is running ta when calling the Xpk library. Option TRUNCATEONPACK: This option is somewhat technical in nature and can be safely ignored. It is only used in case of an error occuring during the compression of a file. In this case, the compressed file has to be deleted, and if this option is set, the handler will try to call SetFileSize() first to truncate the file to 0 bytes (perhaps saving the flushing of a few buffers). However, due to sparse documentation I'm uncertain whether this feature is stable, and hence it is OFF by default. Again, unless you are curious and don't mind risking crashes/data losses, forget about this option. In case of an error during the scanning of the option files the handler will fail its initialisation and hence refuse to load with error code 114 (Bad Template). Sorry, but there are currently no real error messages implented (this will hopefully be fixed in a later version). If the handler refuses to work for no apparent reason, be sure to tripple-check your option files for errors. --- O --- Limitations and known bugs. The error detection code in the initialisation part of the handler is somewhat flaky - I've tried to make it resonably safe, but documentation on the right way to start a handler is hard to find. What it means is that it is a good idea to make sure that the handler is placed in L: and that the nessesary Xpk libraries are placed in libs: before starting the handler (remember, XFH won't be able to tell you the reason if it was unable to initialise for some reason). Another subtle problem is that due to a quirk of the device list locking, it is vital (using XFH v1.12) that the handler for the directory that XFH is to sit in is already loaded. Usually this will not be a problem; however, if you are using 3rd party filesystem handlers that are mounted after boot-up, you can avoid problems by accessing them before mounting XFH (for example by creating an assign (not late-binding) to them). It should be noted that the way the XFH makes the same files available by two different routes is not without its problems. One problem is connected to the volume name - XFH tries to be smart about it, but it will sometimes create a duplicate volume name which is a bad idea. To solve this problem the VOLUMENAME option should be used in an option file. Another problem when mounting XFH in the root dir appears when using the 'Leave Out' feature of the 2.0 Workbench. Here, the '.backdrop' is duplicated in both volumes, making the left-out icons appear twice. I'm working on a decent solution to this problem. Meanwhile, I would recommend that XFH is mounted only on top of subdirectories. For convenience, the handler allows the opening of and writing to new files. Also, XFH now facilitates automatic compression, as discussed in the description of the AUTOCOMPRESS option. However, note that the handler does not yet support opening existing files for writing (this is technically refered to as MODE_READWRITE or ACTION_FINDUPDATE), and writing to compressed files opened in shared mode (MODE_OLDFILE) also does not work. This means that for example it is not possible to append something to a file using the shell '>>' redirection. Still, keeping this in mind it should nevertheless be possible to use the handler even in directories where files are often being written to. Needless to say (but I'll say it anyway), you should not assign libs: to a XFH unit unless you are absolutely sure of what you are doing. A nice trap is to have the XFH call (and wait for) Xpk, which will then wait for XFH to load a particular library for it... This also applies to s: (since the option files are stored here). It should be noted that any given unit of the XFH binds to a directory, not to a DOS device. This means that, currently, it is not possible to have a XFH unit working like DFx: - any access will refer to the disk that was in the drive when the handler was started, not to the disk currently in the drive. The figures reported by the shell 'Info' command are somewhat strange. The problem is that it isn't really possible to give sensible figures for 'NumBlocks' and 'NumBlocksUsed'. Currently, their values are the same as those for the underlying file system. XFH has problems with exclusive locks (for example trying to obtain an exclusive lock on "/" or "foo//" will always fail). Some people have experienced problems when using XFH with the arp.library. This is because a bug/feature in the apr.library function CompareLock() (it declares two locks equal only if the lock->fl_Key fields are equal, which is illegal according to the 2.0 DOS manual). Under 2.0, the program in the directory 'patcharp' can be used to patch arp to use the correct 2.0 SameLock() call. Also, I know of a program that remaps the arp calls to the corresponding 2.0 dos.library calls (though I haven't tried it), it might help too. (This one is a bit technical in nature:) Currently, for reasons having to do with the reading of the primary option file, 'S:' must to refer to an assign (or a volume), not to a device like df0: or ram:. However, since it is almost always an assign (to SYS:s or something), this shouldn't be a big problem. It has been reported to me that some disk toolkit programs (like DiskX) have problems with the "dummy.devide" entry in the mountlist. If you experience any problems with this, you should try changing this entry to something else (like "trackdisk.device" or "scsi.device") or omit it entirely; it is not used by XFH and is only there for the sake of the 'Mount' command. --- O --- Theory of operation. XFH works by installing itself in the system as a file system handler like DF0: or RAM:. However, unlike most file system handlers, which sit on top of a device (or rest in themselves like RAM:), XFH sits on top of an underlying file system handler (abbrevated to UFS) containing a mixture of normal and compressed files. After initialising itself, XFH sits in a loop receiving packets from AmigaDOS and other applications. Each packet is examined and the appropriate action taken. For example, an open request will cause XFH to open the given file, check whether it is compressed, and if so unpack it to memory for later read requests. XFH is currently single-threaded, unlike the Commodore file systems (this means that it is not possible for the XFH to service other requests while waiting for the UFS). I'm hoping to fix this in some later version. --- O --- Acknowledgments. The author wishes to thank all the people that have participated in the development of Xpk without which the XFH would not have been the same. I am especially grateful to Urban D. Müller for helping me start the whole concept of the XFH back in the summer of 1991 - without his help the XFH is not likely to have been realised. Thanks also to the many beta testers who helped me iron out as many bugs as possible before release; your help work has been very valuable to me. And thanks must go, of course, to the guys at Commodore for bringing to us the wonderful Amiga.